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AN  EXAMINATION  OF  THE  WORKLOAD  CONDITIONS 
ASSOCIATED  WITH  OPERATIONAL  ERRORS/DEVIATIONS 
AT  AIR  ROUTE  TRAFFIC  CONTROL  CENTERS 

David  |.  Schroeder  and  Lendell  G.  Nye 


INTRODUCTION 

In  addition  to  the  analysis  of  aircraft  accidents,  several 
other  aspects  of  aviation  activity  are  monitored  and 
evaluated  by  the  Federal  Aviation  Administration  (FAA) 
to  assess  the  overall  safety  of  the  National  Airspace 
System  (NAS).  This  study  involves  the  analysis  of data 
for  one  such  measure,  Operational  Errors/Deviations 
(OEDs).  An  operational  error  (OE)  takes  place  when 
an  air  traffic  controller  allows  less  than  applicable 
minimum  separation  criteria  between  aircraft  (or  an 
aircraft  and  an  obstruction).  Standards  for  separation 
minima  are  described  in  the  AirTraffic  Control  (ATC) 
Handbook  (71 10.65,  and  supplemental  instructions). 
In  the  context  of  the  Operational  Error/Deviation 
(OED)  reporting  system,  an  operational  deviation 
occurs  when  the  appropriate  separation  standards  are 
maintained,  but  an  aircraft  enters  airspace  assigned  to 
another  controller’s  position  of  operation  or  air  traffic 
control  facility  without  prior  approval. 

Since  1985,  the  FAA  has  operated  the  OED  report¬ 
ing  system  to  track  the  operational  errors/deviations 
reported  across  the  nation’s  ATC  system.  An  auto¬ 
matic  OE  detection  system  was  implemented  in  do¬ 
mestic  Air  Route  Traffic  Control  Centers  (ARTCCs) 
in  1984.  Under  this  system,  an  automatic  alarm  is 
triggered  whenever  minimum  separation  standards  are 
violated  by  radar-tracked  aircraft.  When  an  OED 
occurs,  the  associated  facility  is  responsible  for  gather¬ 
ing  data  and  completing  a  report,  in  accordance  with 
FAA  Order  7210.3  (Facility  Operation  and  Adminis¬ 
tration).  Details  concerning  the  air  traffic  control 
situation  and  the  involved  controller(s)  are  gathered 
and  summarized  on  the  reporting  form.  Data  gathered 
via  this  reporting  program  are  then  coded  and  entered 
into  the  OED  computerized  data  base,  under  the 
purview  of  the  Office  of  Aviation  Safety.  The  current 
OEDS  data  base  contains  data  from  the  time  of  its 


implementation  in  1 985.  This  data  base  contains  OED 
data  from  Terminal  facilities  (radar  approach  control 
and  towers)  and  ARTCCs. 

This  study  will  focus  on  data  gathered  from 
ARTCCs,  also  called  En  Route  facilities;  Air  Traffic 
Control  Specialists  (ATCSs)  at  those  facilities  prima¬ 
rily  handle  aircraft  traveling  between  the  terminal 
facilities  across  the  nation.  The  study  was  designed  to 
examine  relationships  between  workload  (number  of 
aircraft  and  complexity  of  air  traffic)  and  causal  factors 
associated  with  ARTCC  OEDs.  First,  this  study 
examined  the  relationship  between  the  two  workload 
measures  for  ail  ARTCCs  combined  and  determined 
whether  there  were  significant  differences  between 
facilities  in  either  the  average  number  of  aircraft  being 
worked  or  the  complexity  level  reported  at  the  time  of 
the  OED  occurrence.  Using  previous  OED  reporting 
systems,  the  results  of  analyses  of  1965  and  1966  data 
(Kershner,  1968),  data  from  Kinney,  Spahn  and  Amato 
(1977),  and  subsequent  operational  error  data 
Schroeder  (1982),  indicated  a  general  trend  for  an 
increased  percentage  oferrors  under  “Light”  and  “Mod¬ 
erate,”  as  compared  to  “Heavy”  workloads.  Stager  and 
Hameluck  (1990)  called  for  an  examination  of  the 
causes  of  operational  errors  in  terms  of  the  conditions 
existent  for  the  air  traffic  controller  during  the  opera¬ 
tive  time-frame  of  the  incident.  These  conditions 
included  “fatigue,  distraction,  attitudes,  excessive 
workload,  and  procedural  knowledge.”  Second,  this 
study  determined  if  certain  categories  of  operational 
errors  occur  more  or  less  frequently  under  the  different 
workload  conditions.  Sperandio  (1971)  found  that  air 
traffic  controllers  tended  to  adopt  different  methods  of 
operation  in  response  to  alterations  in  workload.  Third, 
the  number  of  OEDs  at  a  given  ARTCC  has  been 
found  (Splawn,  Edwards,  &  Chin,  1988)  to  be  a 
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function  of  a)  primarily,  the  total  number  of  opera¬ 
tions  and  b)  secondarily,  the  percentage  of  ATCSs  at 
the  facility  who  have  reached  full  performance  level 
(FPL).  This  finding  for  calendar  year  1987  was  evalu¬ 
ated  for  consistency  over  the  entire  1985-88  time 
period  and  was  expanded  to  include  the  facility’s 
average  complexity  of  air  traffic  reported  during  the 
OED  occurrence  as  an  additional  potential  predictor 
of  the  number  of  OEDs  for  the  ARTCC  facilities. 

METHOD 
OEDS  Data  Base 

Information  was  extracted  from  the  data  base  for  all 
1985-88  ARTCC  OEDs.  The  “number  of  aircraft” 
variable  refers  to  the  number  of  aircraft  that  were  being 
worked  (radar  identified  and/or  established  radio 
communication)  at  the  time  of  the  OED  occurrence. 
Traffic  complexity  involves  the  use,  by  the  quality 
assurance  specialist,  of  a  five-point  rating  scale  (traffic 
was  1-  “easy”  to  5-  “complex”)  to  indicate  the  “overall 
difficulty  of  the  controller’s  task  considering  weather, 
traffic  mix,  variety  of  operations,  limited  use  of  altitudes, 
airspace  available  for  radar  vectoring,  coordination 
requirements,  etc.”  (FAA  Form  7210-3).  It  should  be 
noted  that  the  instructional  guide  for  the  form  differed 
from  the  OED  report  form  in  referring  to  the  initial 
end  point  as  “not  complex”  and  the  final  as  “extremely 
complex.”  Workload  information  was  provided  by  the 
facility  in  the  final  reports.  However,  final  reports  were 
not  required  for  all  OEDs  that  were  classified  as 
“minor.”  Thus,  only  the  final  reports  with  complete 
controller  workload  information  were  included  in  this 
study. 

The  OED  data  base  lists  the  types  of  “causal  factors” 
involved  in  each  incident.  The  causal  factors  for 
ARTCCs  were  categorized  as;  a)  radar  display  factor  - 
inappropriate  use  of  displayed  data,  misidentification, 
etc.;  b)  communication  factor  -  improper  phraseology, 
readback,  etc.;  c)  coordination  factor  -  between  sec¬ 
tors,  facilities,  etc.;  d)  data  posting  -  computer  entry 
and  processing  flight  progress  strips;  and  e)  position 
relief  briefings  -  deficiencies  in  giving  or  utilizing 
briefings.  An  error  could  involve  more  than  one  em¬ 
ployee  and  have  multiple  contributing  causes;  there¬ 
fore,  for  this  study,  an  error  was  considered  to  have 


involved  a  given  factor  if  it  was  identified  as  a  contrib¬ 
uting  factor  for  one  or  more  employees.  Appendix  A 
provides  the  definitions  of  the  causal  factors  from  the 
Final  Operational  Error/Deviation  Report  (FAA  Form 
7210-3). 

Analyses 

Means  and  standard  deviations  were  determined  for 
the  two  workload  measures  for  each  year.  For 
comparisons  across  facilities,  the  “number  of  aircraft” 
measure  was  separated  into  three  categories;  1  -6  aircraft, 
7-11  aircraft,  and  12+  aircraft.  Percentages  of  OEDs 
that  occurred  under  three  levels  of  complexity  and 
“number  of  aircraft  worked”  conditions  were 
determined  for  each  ARTCC.  Causal  factors  were 
coded  as  “0”  if  the  factor  was  not  involved  in  the  error 
and  “  1  ”  if  the  factor  was  listed  as  having  been  involved 
one  or  more  times  in  the  report.  Analyses  were  confined 
to  those  OED  records  with  at  least  one  ATCS  employee 
causal  factor  and  non  missing  data  for  (a)  the  number 
of  aircraft  being  worked  and  (b)  traffic  complexity. 
Correlation  coefficients  were  computed  between  the 
workload  measures  and  error  factors.  Multiple 
regression  analyses  were  performed  to  determine  the 
relationships  of  several  average  facility-level  conditions 
with  the  number  of  OEDs  for  the  ARTCCs. 

RESULTS 

The  average  number  of  aircraft  worked  during  an 
OED  declined  slightly  from  1985  to  1988:  Af=9.6, 
5Z>=4.7  (1985);  M=9.2,  SD=4.5  (1986);  A/=8.6, 
5D=4.1  (1987);  and  M=8.7,  SD=4.2  (1988).  This  was 
true  despite  the  general  increase  in  the  overall  number 
of  aircraft  operations  handled  by  the  ARTCCs,  from 
approximately  32.27  million  in  1985  to  35-91  million 
in  1 988.  Percentages  of  OEDs  occurring  under  the  five 
complexity  levels  for  each  year  are  presented  in  Table 
1. 

While  the  percentages  of  occurrences  for  the  differ¬ 
ent  complexity  levels  were  similar  for  1985  and  1986, 
in  the  latter  two  years,  there  was  a  slight  increase  in  the 
percentage  of  errors  that  occurred  under  “easy”  condi¬ 
tions  (9.8%  in  1987  and  10.7%  in  1988  versus  4.8% 
and  4.6%  in  1985  and  1986,  respectively).  Also,  across 
the  four  years,  from  21%  to  29%  occurred  under  less 
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Table  1 

Percentages  of  OEDs  Occurring  Under  each  of  Five  Complexity  Conditions,  by  Year 


YEAR 

EASY 

AVERAGE 

COMPLEX 

1985 

4.8 

18.3 

39.7 

28.6 

8.6 

1986 

4.6 

16.7 

40.1 

29.9 

8.6 

1987 

9.8 

15.6 

37.8 

28.2 

8.5 

1988 

10.7 

18.6 

38.0 

27.3 

5.5 

Overall 

7.4 

17.3 

38.9 

28.5 

7.9 

Note;X2(12)=30.83;  p<.01 


Table  2 

Number  of  Aircraft  Worked  During  OEDs  for  Traffic  Complexity  Categories 


TRAFFIC  COMPLEXITY 

/VOF 

OEDS 

MEAN 

#  AIRCRAFT 

SD 

MINIMUM 

MAXIM 

Easy 

155 

4.01 

2.0 

1 

13 

Below  Average 

356 

6.03 

2.3 

1 

17 

Average 

800 

8.66 

3.2 

1 

25 

Above  Average 

576 

11.41 

4.1 

3 

35 

Complex 

161 

14.30 

5.6 

5 

40 

Totals 

2048 

9.07 

4.5 

1 

40 

than  average  complexity  conditions.  Furthermore, 
nearly  two-thirds  of  the  errors  occurred  under  average 
or  lower  complexity.  The  interrelationship  between 
the  two  measures  is  demonstrated  in  Table  2.  Even 
though  there  is  a  gradual  increase  in  the  average  num¬ 
ber  of  aircraft  worked  under  the  five  complexity  levels, 
from  4.01  under  “easy”  conditions  to  14.30  under 
“complex”  conditions,  the  range  within  each  of  the 
categories  is  considerable.  “Easy”  conditions  involved 
from  1  to  13  aircraft  and  “complex”  conditions  in¬ 
volved  from  5  to  40  aircraft.  (Note:  involvement  of  a 
single  aircraft  occurred  in  only  1 1  of  the  incidents,  a 
majority  of  which  were  operational  deviations).  Dif¬ 
ferences  across  the  five  categories  and  the  general 
overlap  would  seem  to  raise  some  questions  concern¬ 


ing  the  criteria  used  to  determine  air  traffic  complexity 
and  whether  those  criteria  are  consistently  applied 
across  facilities.  Under  what  circumstances  would  a 
situation  involving  3  aircraft  be  of  “above  average” 
complexity  for  a  particular  sector?  Likewise,  how  can  a 
situation  involving  25  aircraft  be  considered  to  be  of 
only  “average"  complexity?  Additional  information 
and  analyses  are  required  to  determine  the  factors  most 
closely  related  to  application  of  the  complexity  ratings 
and  whether  those  ratings  should  be  modified. 

Percentages  of  OEDs  occurring  under  “less  than 
average”  (“easy”  and  “below  average  complexity”), 
“average,”  and  “above  average”  complexity  (“above 
average”  and  “complex”)  for  each  of  22  ARTCCs  are 
presented  in  Table  3.  The  Honolulu  (ZHN)  ARTCC 
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Table  3 

Percentage  of  OEDs  Occurring  Under  Three  Complexity  Conditions  for  each  ARTCC 


COMPLEXITY  RATING 


LESS  THAN 

AVERAGE 

ABOVE 

ARTCC 

AVERAGE 

COMPLEXITY 

AVERAGE 

tl 

ZAB  - 

Albuquerque 

27.3% 

40.9% 

31.8% 

44 

ZAN  - 

Anchorage 

15.4% 

34.6% 

50.0% 

26 

ZAU  - 

Chicago 

29.1% 

45.6% 

25.3% 

158 

ZBW- 

Boston 

1 8.5% 

53.3% 

28.1% 

135 

ZDC  - 

Washington  DC 

21.1% 

29.7% 

49.2% 

185 

ZDV  - 

Denver 

17.9% 

29.5% 

52.6% 

78 

ZFW  - 

Fort  Worth 

1 6.9% 

39.8% 

43.4% 

83 

ZHU  - 

Houston 

28.2% 

37.6% 

34.1% 

85 

ZID  - 

Indianapolis 

20.0% 

49.0% 

31.0% 

155 

ZJX  - 

Jacksonville 

26.4% 

44.8% 

28.7% 

87 

ZKC  - 

Kansas  City 

27.8% 

30.6% 

41.7% 

108 

ZLA  - 

Los  Angeles 

29.4% 

24.6% 

46.0% 

126 

ZLC  - 

Salt  Lake  City 

27.0% 

35.1% 

37.8% 

37 

ZMA- 

Miami 

23.8% 

54.8% 

21.4% 

42 

ZME  - 

Memphis 

22.4% 

44.7% 

32.9% 

76 

ZMP  - 

Minneapolis 

26.8% 

49.3% 

23.9% 

71 

ZNY  - 

New  York  City 

27.9% 

36.6% 

35.5% 

172 

ZOA- 

Oakland 

25.4% 

40.8% 

33.8% 

71 

ZOB  - 

Cleveland 

31.1% 

32.0% 

36.9% 

122 

ZSE  - 

Seattle 

29.0% 

29.0% 

41.9% 

31 

ZSU  - 

San  Juan 

36.4% 

21.2% 

42.4% 

33 

ZTL  - 

Atlanta 

25.4% 

46.5% 

28.1% 

114 

Combined 

25.0% 

39.1% 

35.9% 

2039 

Note:X*(42)=98.42;  p<.001 


was  not  included,  due  to  the  small  number  of  OEDs 
(less  than  10)  in  the  database.  There  were  marked 
differences  across  facilities  in  the  percentages  of  OEDs 
occurring  under  the  three  complexity  levels.  While 
25%  of  the  OEDs  occurred  under  “less  than  average” 
complexity  categories,  the  percentages  across  facilities 
ranged  from  15.4%  at  the  Anchorage  ARTCC  to 
31.1%  at  Cleveland  and  36.4%  at  San  Juan.  Similar 
differences  were  noted  for  “average”  complexity  (rang¬ 
ing  from  21.2%  at  San  Juan  to  54.8%  at  Miami). 
Percentages  of  OEDs  under  “above  average”  complex¬ 
ity  ranged  from  21.4%  at  Miami  to  52.6%  at  Denver. 
Of  the  22  facilities,  four  (ZAN,  ZDC,  ZDV,  and  ZLA) 
had  more  than  45%  of  their  OEDs  occur  under  “above 


average”  complexity  conditions.  In  contrast,  three 
(ZAU,  ZMA,  and  ZMP)  had  26%  or  less  of  their 
OEDs  occur  under  “above  average”  complexity  condi¬ 
tions.  The  extent  to  which  these  differences  reflect 
actual  differences  in  complexity  of  the  sectors  at  the 
various  ARTCCs,  or  to  differences  in  how  the  com¬ 
plexity  ratings  are  assigned,  is  not  determinable  from 
the  available  data.  Certainly  within  each  facility,  it  is 
reasonable  to  expect  that  some  sectors  present  more 
complex  and  challenging  air  traffic  control  situation;, 
than  others.  Also,  a  sector  considered  “complex”  at  one 
facility  might  be  rated  as  “average”  at  another  ARTCC. 
Thus,  without  additional  information  concerning  the 
percentage  of  time  ATCSs  at  the  various  facilities 


Tabie  4 

Percentage  of  OEDs  Occurring  Under  each  of  Three  Workload  Categories  for  each  ARTCC 


NUMBER  OF  AIRCRAFT 


1-6 

7-11 

12  OR  MORE 

ARTCC 

AIRCRAFT 

AIRCRAFT 

AIRCRAFT 

ZAB  - 

Albuquerque 

22.7% 

50.0% 

27.3% 

44 

ZAN  - 

Anchorage 

23.1% 

1 1 .5% 

65.4% 

26 

ZAU  - 

Chicago 

31.0% 

43.7% 

25.3% 

158 

ZBW- 

Boston 

45.2% 

43.7% 

11.1% 

135 

ZDC  - 

Washington  DC 

25.9% 

49.7% 

24.3% 

185 

ZDV  - 

Denver 

15.4% 

44.9% 

39.7% 

78 

ZFW- 

Fort  Worth 

33.7% 

51.8% 

14.5% 

83 

ZHU  - 

Houston 

38.8% 

43.5% 

1  7.6% 

85 

ZID  - 

Indianapolis 

29.7% 

58.1% 

12.3% 

155 

ZJX  - 

Jacksonville 

21.8% 

43.7% 

34.5% 

87 

ZKC  - 

Kansas  City 

30.6% 

55.6% 

13.9% 

108 

ZLA  - 

Los  Angeles 

53.2% 

39.7% 

7.1% 

126 

ZLC  - 

Salt  Lake  City 

13.5% 

32.4% 

54.1% 

37 

ZMA- 

Miami 

16.7% 

66.7% 

16.7% 

42 

ZME  - 

Memphis 

19.7% 

55.3% 

25.0% 

7b 

ZMP  - 

Minneapolis 

22.5% 

49.3% 

28.2% 

71 

ZNY  - 

New  York  City 

39.0% 

32.6% 

28.5% 

172 

ZOA  - 

Oakland 

29.6% 

36.6% 

33.8% 

71 

ZOB  - 

Cleveland 

30.3% 

41.0% 

28.7% 

122 

ZSE  - 

Seattle 

41.9% 

32.3% 

25.8% 

31 

ZSU  - 

San  Juan 

15.2% 

36.4% 

48.5% 

33 

ZTL  - 

Atlanta 

32.5% 

39.5% 

28.1% 

114 

Combined 

31.1% 

44.8% 

24.0% 

2039 

Note:  W2)=1 99.45;  p<.001 


spend  controlling  traffic  under  the  various  complexity 
or  workload  conditions  it  is  difficult  to  determine  the 
primary  factors  associated  with  these  outcomes. 

Differences  in  percentages  of  OEDs  occurring  un¬ 
der  different  reported  workload  conditions  were  also 
evident  in  the  three  categories  of  “number  of  aircraft 
worked”  (Table  4).  Six  facilities  (ZAN,  ZDV,  ZjX, 
ZLC,  ZOA,  and  ZSU)  had  more  than  30%  of  their 
OEDs  occur  when  “12  or  more”  aircraft  were  being 
worked.  ZBW,  ZFW,  ZID,  ZKC  and  ZLA  had  less 
than  15%  of  their  OEDs  occur  under  this  higher 
workload  condition.  Three  ARTCCs  (ZBW,  ZLA, 
and  ZSE)  had  more  than  40%  of  their  OEDs  occur 


under  the  lower  traffic  condition  (6  aircraft  or  less). 
Once  again,  additional  information  is  needed  to  clearly 
understand  the  basis  for  these  differences.  To  what 
extent  are  the  differences  related  to  how  determina¬ 
tions  are  made  concerning  the  number  of  aircraft 
handled  or  to  aspects  of  the  underlying  traffic  condi¬ 
tions? 

Correlations  between  the  two  workload  measures 
“number  of  aircraft”  (#WRKD)  and  complexity 
(CMPLXTY)  and  the  five  causal  factor  categories  are 
presented  in  Table  5.  Z#WRKD  refers  to  the  value  of 
the  “number  of  aircraft”  measure,  following  transfor¬ 
mation  into  a  z  score,  standardized  within  each  facil- 
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Tabu  5 

Intercorrelations  Between  Workload  Measures  and  Causal  Factors 


Z#WRKD 

#WRKD 

CMPLXTY 

DATAP 

RADARD 

COMM  COORD 

Z#WRKD 

— 

#WRKD 

.92** 

— 

CMPLXTY 

.65** 

.61** 

— 

DATAP 

.11** 

.16** 

.06* 

— 

RADARD 

.02 

-.01 

.06* 

-.13** 

— 

COMM 

-.01 

-.01 

-.01 

-.10** 

1 

o 

« 

• 

— 

COORD 

-.05 

-.04 

1 

b 

Kjy 

• 

.16** 

-.23** 

-.15**  — 

RBRIEF 

.05 

.09** 

.03 

.15** 

-.06* 

-.06*  .08** 

Note: 

n  =  2042,  2-tailed  significarce  levels 

*  =  p<.01 

••  =  p<.001 

Z#WRKD  =  number  of  aircraft  being  worked-standardized  within  each  facility 

#WRKD  =  actual  number  of  aircraft  being  worked 

CMPLXTY  =  air  traffic  complexity  rating 

DATAP  =  data  posting  was  a  causal  factor 

RADARD  =  radar  display 

COMM  =  communication 

COORD  =  coordination 

RBRIEF  =  relief  briefing 


ity,  rather  than  on  all  facilities  combined.  The  five 
causal  factors  were:  data  posting  (DATAP);  radar  dis¬ 
play  (RADARD);  communications  (COMM);  coordi¬ 
nation  (COORD);  and  relief  briefing  (RBRIEF).  As 
already  indicated,  complexity  and  ‘number  of  air¬ 
craft”  are  highly  correlated  (r=.6l).  “Number  of  air¬ 
craft”  is  most  closely  related  to  the  data  posting  causal 
factor  (r=.l6,  ^.001).  A  statistically  significant  posi¬ 
tive  correlation  was  also  noted  between  “number  of 
aircraft”  and  the  relief  briefing  factor  (r=.09,^<.001). 
Thus,  OEDs  involving  a  greater  number  of  aircraft 
were  somewhat  more  likely  to  involve  a  problem  asso¬ 
ciated  with  data  posting.  They  were  also  slightly  more 
likely  to  involve  problems  associated  with  the  relief 
briefing.  Complexity  also  appears  to  have  very  small 
statistical  relationships  with  data  posting  (r^. 06,^. 0 1 ) 
and  use  of  the  radar  display  (^^=.06,  /<.0 1 ).  A  small  but 
statistically  significant  negative  correlation  was  noted 


between  complexity  and  the  coordination  factor 
(r=-.05,  ^.0 1 ).  Thus,  an  error  involving  coordination 
was  somewhat  more  likely  to  occur  under  lower,  rather 
than  higher,  complexity  conditions.  While  statistically 
significant,  the  practical  significance  of  these  results  is 
limited,  due  to  the  low  magnitude  of  the  correlation 
coefficients.  Significant  patterns  of  intercorrelations 
were  found  among  types  of  causal  factors  within  the 
OED  incidents.  In  particular,  when  the  radar  display 
was  involved,  communication  and  coordination  fac¬ 
tors  tended  not  to  be  involved  (r=-.40,  /x.OOl  and  r=- 
.23,  ^<.001  respectively). 

To  provide  a  better  understanding  of  the  relation¬ 
ship  between  the  “number  of  aircraft”  workload  mea¬ 
sure  and  the  various  causal  factors,  the  workload 
measure  for  “number  of  aircraft”  was  categorized  on 
the  basis  of  the  four  z  score  ranges:  low  through  -1.0; 
-.99  through  .99;  1.0  through  1.99;  and  2.0  and 


Tabu  6 

Causal  Factor  Types  and  Number  of  Aircraft  Worked 
Categories  Standardized  Within  ARTCC  Facilities 


CAUSAL  FACTOR  #  OF  OEDS 


#  OF  AIRCRAFT 

DATA 

RELIEF 

RADAR 

COMMUNI¬ 

COORDIN¬ 

IN  WORKLOAD 

(Z  VALUES) 

POSTING 

BRIEFING 

DISPLAY 

CATION 

ATION 

LEVEL 

s  -1.0 

15.9% 

2.4% 

49.3% 

31.0% 

42.1% 

n=46 

n=7 

n=143 

o 

II 

c 

n=122 

290 

-.99  -  +.99 

18.6% 

4.0% 

58.3% 

29.7% 

2/.  5% 

n=268 

n=57 

n=839 

n=427 

n=395 

1438 

1.0  -  1.99 

30.3% 

6.1% 

56.7% 

28.6% 

25.1% 

n=70 

n=1 4 

n=131 

n=66 

n=58 

231 

2.0  and  > 

37.1% 

10.1% 

56.2% 

29.2% 

37.1% 

n=33 

n=9 

n=50 

n=26 

n=33 

89 

Overall 

20.4% 

4.2% 

56.8% 

29.7% 

29.6% 

n=417 

n=87 

n=1163 

n=609 

n=608 

2048 

Chi-square 

35.7 

12.1 

8.0 

0.4 

29.3 

idf=3) 

p<.001 

p<.01 

p<.05 

p<.001 

Note;  Results  of  separate  chi-square  tests  of  OEDs  that  involved  a  given  causal  factor  versus 
those  that  did  not  involve  that  factor  (i.e.  all  others).  Row  percentage  totals  exceed  100% 
because  an  OED  can  involve  multiple  causal  factors.  /7=number  of  OEDs  that  occurred  under 
each  combination  of  workload  level  and  causal  factor  type.  %=percentage  of  OEDs  within  a 
workload  level  involving  that  causal  factor. 


greater.  Even  though  the  relationships  between  workload  conditions.  The  Radar  Display  factor  was 

workload  measure.,  and  the  causal  factors  were  low,  the  marginally  less  likely  to  occur  in  the  lowest  workload 

results  were  consistent  with  expectations.  The  percent-  conditions,  and  was  involved  in  from  49%  to  58%  of 

ages  shown  in  Table  6  represent,  within  a  given  the  OEDs.  Coordination  tended  to  be  involved  more 

workload  condition,  the  number  of  OEDs  in  which  a  frequently  at  both  low  and  high  workload  conditions, 

causal  factor  was  involved,  divided  by  the  total  number  compared  to  moderate  or  average  conditions.  Thus, 

of  OEDs  for  that  workload  condition.  It  was  possible  while  causal  factors  were  not  highly  influenced  by 

for  multiple  causal  factors  to  be  listed  for  a  single  OED  either  of  the  two  workload  conditions  (“complexity” 

and  thus  the  number  of  causal  factors  exceed  the  and  “number  of  aircraft”),  there  was  some  evidence 

number  of  OEDs.  Overall,  20.4%  of  the  OEDs  in-  that  higher  percentages  of  certain  error  factors  oc- 

volved  data  posting.  When  a  relatively  large  number  of  curred  at  the  extremes  of  the  workload  distribution, 

aircraft  were  being  handled  (z  score  of  2.0  or  more),  a  rather  than  at  average  to  moderate  workload  condi- 

data  posting  causal  factor  was  reported  in  37.1%  of  tions. 

those  OEDs,  compared  to  15.9%  of  the  OEDs  in  the  Table  7  contains  results  of  a  regression  analysis  in 
lowest  workload  condition.  The  Relief  Briefing  factor  which  the  criterion  -  average  number  of  OEDs  for 
was  also  more  likely  to  be  involved  at  the  higher  ARTCC  facilities  from  1985-88  (EALL)  -  was  evalu- 
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Tabu  7 

Multiple  Regression  for  Operational  Errors/D^vi/tions  at  ARTCCs  from  1985-88 


VARIABLE 

MEAN 

SD 

VARIABLE  LABEL 

FALL 

69.8 

94.9 

Avg.  Annual  ARTCC  Errors 

OPSAVG  1,632,680.29 

442,995.13 

Avg.  Facility  Operations 

FPLRATIO 

.66 

.11 

Ratio  FPL  to  Total  ATCSs 

CMPLXAVG 

3.12 

.16 

Avg  Complexity  Reported  at 
Occurrence  of  OEDs 

Dependent  Variable  = 

FALL  -  Avg.  Annual  ARTCC  Errors 

Multiple  R 

.914 

R  Square 

.836 

Fj  ,,=28.92;  p<.001 

Adjusted  R  Square 

.807 

BETA 

PARTIAL 

VARIABLE 

WEIGHT 

CORR. 

F 

OPSAVG 

.949 

.909 

81.41 

FPLRATIO 

-.332 

-.627 

11.00 

CMPLXAVG 

.301 

.571 

8.22 

ated  as  a  function  of  a)  average  annual  facility  opera¬ 
tions  (OPSAVG),  b)  the  facility’s  ratio  of  full  perfor¬ 
mance  level  (FPL)  controllers  to  total  ATCS  workforce 
(FPLRATIO),  and  c)  the  facility’s  average  traffic  com¬ 
plexity  reported  at  the  time  of  the  occurrence  of  the 
OEDs  (CMPLXAVG).  The  data  for  the  ratio  of  FPL/ 
total  ATCSs  were  obtained  from  the  1987  calendar 
year  report  by  Splawn,  Edwards,  &  Chin  (1988), 
which  approximates  the  mid-point  of  the  time-frame 
of  the  current  study. 

The  set  of  predictor  variables  accounted  for  ap¬ 
proximately  84%  of  the  variance  in  number  of  ART  CC 
OEDs,  with  each  measure  contributing  significantly 
to  the  regression  equation.  The  standardized  beta 
coefficients  and  partial  correlations  were  greatest  for 
the  quantitative  measure  of  facility  workload  (B=.95, 
r=.91)  and  similar  in  magnitude  for  the  FPL  experi¬ 
ence  level  (B=-.33,  r=-.65)  compared  with  average 
traffic  complexity  (B=.30,  r=.57).  As  was  found  for 
1987,  the  average  i  rmber  of  OEDs  during  1985-88 
tended  to  be  strongly  and  positively  related  to  total 
operations  and  inversely  related  to  the  ratio  of  FPL  air 


traffic  controllers.  The  measure  of  the  qualitative  na¬ 
ture  of  a  facility’s  workload  at  the  time  of  the  OEDs 
was  shown  to  be  a  significant  additional  component  in 
explaining  facility  differences  in  the  number  of  OEDs. 
This  last  finding  was  the  result  of  the  inverse  relation¬ 
ship  (r=-.34)  between  an  ARTCC’s  average  operations 
and  average  complexity  (i.e.,  ARTCCs  with  a  greater 
number  of  annual  ATC  operations  tended  to  report 
that  their  OEDs  occurred  under  less  complex  traffic 
conditions).  The  determination  of  whether  or  not 
comparable  facility  differences  exist  in  air  traffic  com¬ 
plexity  during  non-error  conditions  requires  further 
evaluation. 

DISCUSSION 

This  study  found  that  both  the  average  number  of 
aircraft  being  worked  and  the  level  of  the  air  traffic 
complexity  during  OEDs  declined  slightly  during  1 985- 
88.  As  expected,  there  was  a  strong  relationship  between 
the  two  workload  measures  reported  in  the  OED 
database.  Differences  in  the  average  workload 
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conditions  during  the  OEDs  were  found  across  the 
ART  CCs.  Some  degree  of  variability  would  be  expected, 
given  that  the  workload  measures  are  reported  by 
many  different  investigators.  Although  the  report  form 
and  associated  information  provides  detailed 
information  concerning  the  evaluation  process,  certain 
determinations  remain  subjective.  In  particular,  the 
criteria  for  categorizing  air  traffic  complexity  seems  to 
require  additional  study.  It  is  possible  that  an  analysis 
of  comments  provided  in  the  section  of  the  report 
devoted  to  “explanation  of  traffic  complexity”  could 
provide  some  insight  into  differences  in  the  rating 
procedures.  Also,  to  what  extent  and  how  consistently 
are  the  “number  of  aircraft  being  worked  at  the  time  of 
occurrence”  reported  as  the  number  handled  at  the 
time  of  actual  loss  of  separation  or  as  the  number 
handled  during  a  specified  time  period  surrounding 
the  OED? 

Some  causal  factors  were  found  to  occur  more  often 
under  certain  workload  levels.  The  strongest  associa¬ 
tion  between  workload  and  the  type  of  causal  factor 
contributing  to  the  OEDs  was  for  data  posting  -  which 
involves  computer  entry  and  processing  of  flight 
progress  strips.  However,  these  relationships  were  small 
and  tended  to  be  evident  only  at  the  extremes  of  the 
workload  conditions.  These  findings  offer  some  sup¬ 
port  to  Sperandio  (1971),  who  found  that  alterations 
in  workload  led  ATCSs  to  adopt  different  modes  of 
operation.  Also,  the  results  are  consistent  with  recent 
research  regarding  how  air  traffic  controllers  handle 
board  management  under  varying  traffic  scenarios 
(Vortac,  O.U.,  et  al).  A  question  for  future  studv  is 
whether,  and  to  what  extent,  air  traffic  controllers 
implement  less  effective  a)  procedures  in  response  to 
changes  in  workload,  and  b)  strategies  as  a  conse¬ 
quence  of  high  workload? 

The  number  of  incidents  at  ARTCCs  during  the 
calendar  years  1985-88  was  found  to  be  positively 
associated  with  total  number  of  facility  operations,  but 
inversely  related  to  the  ARTCC’s  proportion  of  air 
traffic  controllers  who  were  at  full  performance  level. 
This  relationship  is  consistent  with  previous  research 
on  operational  errors  at  ARTCCs  for  1987  (Splawn, 
Edwards,  &  Chin,  1988).  In  the  current  study,  it  was 
found  that  after  accounting  for  the  total  number  of 
operations  and  FPL  ratio  variables,  a  facility’s  average 


air  traffic  complexity  level  (reported  at  the  time  of  its 
OEDs)  was  also  a  significant  predictor  of  total  OEDs, 
with  84%  of  the  variance  associated  with  the  OED 
data  subsumed  by  those  three  variables.  Thus,  al¬ 
though  most  OEDs  occurred  under  average  or  lower 
air  traffic  complexity,  this  more  qualitative  measure  of 
workload  helped  explain  the  variation  in  the  number 
of  OEDs  at  cn  route  facilities. 
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FACTORS  ASSOCIATED  WITH  THE  SEVERITY  OF  OPERATIONAL 
ERRORS  AT  AIR  ROUTE  TRAFFIC  CONTROL  CENTERS 

Mark  D.  Rodgers  and  Lendell  G.  Nye 


INTRODUCTION 

In  the  history  of  the  Federal  Aviation  Administration, 
no  aircraft  have  collided  while  under  positive  control 
in  en  route  airspace.  However,  aircraft  have  viciuied 
prescribed  separation  minima  and  approached  in  close 
proximity.  This  event  can  occur  as  a  result  of  either  a 
pilot  deviation,  or  an  operational  error  (OE).  In  this 
study,  analyses  were  conducted  of  data  gathered 
concerning  OEs.  An  operational  error  takes  place 
when  an  air  traffic  controller  allows  less  than  applicable 
minimum  separation  criteria  between  aircraft  (or  an 
aircraft  and  an  obstruction).  Standatds  fot  separation 
minima  are  described  in  the  Air  T raffic  Control  (AT C) 
Handbook  (711 0.65F,  and  supplemental  instructions). 
While  thete  is  considerable  complexity  in  those 
standards,  at  flight  levels  between  29,000  and  45,000 
feet.  Air  Traffic  Control  Specialists  (ATCSs)  at  en 
route  facilities  are  required  to  maintain  either  2,000 
feet  vertical  separation  or  5  miles  horizontal  separation 
between  aircraft.  At  flight  levels  below  29,000  feet 
with  aircraft  under  IFR  conditions,  AT CSs  are  required 
to  maintain  either  1,000  feet  vertical  separation  or  5 
miles  horizontal  separation.  This  study  focused  on 
data  gathered  from  Air  Route  Traffic  Control  Centers 
(ARTCCs),  also  called  en  route  facilities. 

The  first  purpose  of  this  study  was  to  examine  the 
relationship  between  the  level  of  severity  (degree  of 
loss  of  vertical  and  horizontal  separation  between 
aircraft)  and  air  traffic  controller  workload,  measured 
by  the  number  of  aircraft  being  worked  and  complex¬ 
ity  level  reported  at  the  time  of  the  occurrence.  The 
analyses  of  1965  and  1966  data  by  Kershner  (1968), 
data  from  Kinney,  Spahn,  and  Amato  (1977),  and 
analyses  of  subsequent  operational  error  data 
(Schroeder,  1982,  Stager  and  Hameluck,  1990,  and 
Schroederand  Nye,  1993)  found  a  general  trend  for  an 
increased  percentage  of  errors  under  “Light”  and  “Mod¬ 
erate,”  as  compared  to  “Heavy”  workloads.  Schroeder 
and  Nye  (1991)  also  found  two  measures  of  workload 


(number  of  aircraft  being  worked  and  situation  com¬ 
plexity)  to  be  highly  correlated  with  each  other.  Al¬ 
though  OEs  tend  to  occur  more  often  under  light  and 
moderate  workloads,  it  was  hypothesized  that  the 
more  severe  OEs  occur  under  heavy  workload  condi¬ 
tions. 

The  second  purpose  was  to  determine  which  of  the 
“causal  factors"  attributed  to  the  air  traffic  controller 
tended  to  be  associated  with  OE  severity.  The  general 
categories  of  causal  factors  include:  Data  Posting, 
Radar  Display,  Coordination,  Communication,  and 
Position  Relief  Briefing.  Each  of  these  factors  is  sub¬ 
divided  into  more  specific  categories,  with  5-15  sub- 
categories  under  each  main  factor.  It  was  hypothesized 
that  the  OEs  involving  the  causal  factor  categories 
more  closely  related  to  the  ATCS’s  situation  awareness 
were  more  likely  to  result  in  greater  severity,  that  is, 
more  serious  separation  errors.  Another  variable  (not 
defined  as  a  causal  factor)  reported  on  the  OE  form 
addresses  whether  or  not  the  ATCS  was  aware  the  error 
was  developing.  This  factor  was  included  in  the  analy¬ 
ses,  since  it  relates  directly  to  the  hypothesis. 

The  third  purpose  of  this  study  was  to  determine  the 
relationship  between  the  severity  of  an  OE  and  two  of 
the  flight  characteristics  of  the  aircraft  involved  in  the 
errors:  aircraft  profile  and  altitude.  It  was  hypoth¬ 
esized  that  aircraft  in  a  climbing  or  descending  profile 
are  more  likely  to  be  involved  in  the  more  severe  OEs, 
due  to  the  rapidly  changing  dynamics  of  the  control 
situation.  It  was  also  hypothesized  that  aircraft  at  lower 
altitudes,  where  less  separation  is  required,  are  more 
likely  to  be  involved  in  the  more  severe  OEs. 

The  fourth  purpose  of  this  study  was  to  determine 
how  these  variables  are  distributed  among  individual 
ARTCCs.  It  was  hypothesized  that  clusters  of  facilities 
could  be  determined  in  regard  to  certain  factors  related 
to  OE  severity. 


The  fifth  purpose  of  the  study  was  to  determine 
what  factors  operated  during  a  “major”  OE.  Since 
there  were  only  15  major  OFs  in  the  database  during 
the  time-frame  of  this  study,  only  frequencies  of  the 
factor  categories  occurring  during  a  major  error  were 
reviewed. 

METHOD 

Operational  Error  Data  Base 

Since  1985  the  FAA  has  operated  an  Operational  Error 
Data  Base  to  track  the  operational  errors  reported 
across  the  nation’s  ATC  system.  Quality  Assurance 
personnel  at  each  facility  are  responsible  for  gathering 
data  and  completing  a  report  in  accordance  with  FAA 
Order  72 10.3  (FacilityOperation  and  Administration). 
Data  are  coded  and  entered  into  the  operational  error 
computerized  data  base,  which  is  under  the  purview  of 
the  Office  of  Aviation  Safety.  Since  1988,  the  data  base 
has  included  a  more  detailed  encoding  of  the  “causal 
factor”  information.  Therefore,  for  the  purposes  of 
this  report,  the  sample  includes  the  error  reports  for 
the  years  1988,  1989,  1990,  and  approximately  two- 
thirds  of  the  OEs  from  1991,  which  included 
information  on  ATCS  causal  factors  and  workload, 
along  with  OE  severity. 

Error  Severity 

The  severity  of  an  OE  is  categorized  and  reported  by 
the  FAA  according  to  the  closest  proximity  of  aircraft, 
in  terms  of  both  horizontal  and  vertical  distances. 
Table  1  shows  the  separation  parameters  and 
corresponding  point  assignments,  which  are  added  to 
determine  whether  the  severity  was  “major,” 
“moderate,”  or  “minor.”  This  calculation  is  made  by  a 
sub-routine  of  the  OE  data  base.  Of  the  1053  errors  in 
our  sample,  only  15  were  coded  as  “major”  errors, 
defined  as  being  less  than  0 . 5  miles  horizontal  separation 
AND  less  than  500  feet  vertical  separation.  Meeting 
this  criterion  is  the  only  way  that  an  error  can  be  rated 
“major,”  whereas  there  are  many  ways  that  an  error  can 
be  rated  “minor”  or  “moderate.”  Some  of  the  key 
points  regarding  the  separation  standards  can  be 
summarized  as  follows: 


a)  a  “major”  error  is  defined  as  less  than  0.5  miles 
horizontal  separation  AND  less  than  500  feet 
vertical  separation  between  aircraft  at  the  time  of 
occurrence; 

b)  the  designation  of  “moderate”  or  “minor"  error 
is  based  on  the  determination  of  the  altitude  of 
occurrence: 

1)  at  29,000  ft  (FL  290)  or  below,  a  moderate 
error  can  involve  a  range  of  separation  param¬ 
eters  from  less  than  0.5  miles  horizontal  com¬ 
bined  with  up  to  900  feet  vertical  at  one 
extreme,  compared  to  almost  3.0  miles  hori¬ 
zontal  combined  with  less  than  500  feet  verti¬ 
cal.  A  minor  error  can  involve  less  than  0.5 
miles  horizontal  separation  only  if  vertical 
separation  is  900  feet  or  greater.  Also,  an  error 
can  be  classified  as  minor  with  less  than  500 
feet  vertical  separation  only  if  the  aircraft  were 
3.0  miles  or  farther  apart. 

2)  above  FL  290,  a  moderate  error  can  involve 
less  than  0.5  miles  horizontal  combined  with 
up  to  1000  feet  vertical  (compared  to  900  ft. 
at  FL  290  or  lower).  At  this  flight  level,  while 
severity  points  are  added  for  vertical  separa- 
'■on  up  to  2,000  feet,  all  errors  occurring  with 
vertical  separation  of  1 ,000  ft.  or  greater  are 
classified  as  minor. 

Of  the  15  “major”  errors,  7  resulted  in  the  filing  of 
a  near  mid-air  collision  report  by  at  least  one  of  the 
involved  pilots.  Since,  for  the  above  reasons,  the  FAA 
has  categorized  these  errors  as  qualitatively  different 
from  the  other  errors,  and  because  there  were  so  few  of 
them,  they  were  analyzed  separately,  with  only  fre¬ 
quency  data  presented  in  this  report. 

Air  Traffic  Controller  Workload  and  Causal 
Factors 

The  two  measures  of  workload  reported  in  the  OE  data 
base  are  the  number  of  aircraft  being  worked  by  the  air 
traffic  controller  at  the  time  of  the  error  and  a  rating  of 
the  air  traffic  complexity,  which  is  an  estimation  by  the 
quality  assurance  specialist  of  the  difficulty  of  the  job 
tasks,  based  on  factors  such  as  weather,  airspace 
restrictions,  and  variety  of  duties.  The  complexity 
rating  is  made  on  a  5-point  scale  with  anchors  1  = 
easy,  3  =  average,  and  5  =  complex. 
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Table  1 

Definition  of  Severity  Categories 


VERTICAL  SEPARATION  POINTS 


If  occurrence  was  at  or  below 
altitude  of  29,000  ft.  (FL  290): 

Less  than  500  feet . 1 0 

500  feet  to  600  feet . 9 

600  feet  to  700  feet . 8 

700  feet  to  800  feet . 6 

800  feet  to  900  feet . 4 

900  feet  to  1 ,000  feet . 2 

If  occurrence  was  above  altitude 
of  29,000  ft.  (FL  290): 

Less  than  500  feet . 1 0 

500  feet  to  700  feet . 9 

700  feet  to  1 ,000  feet . 8 

1 ,000  feet  to  1 ,500  feet . 3 

1 ,500  feet  to  2,000  feet . 2 


HORIZONTAL  SEPARATION  POINTS 

Less  than  'A  mile . 1 0 

'h  mile  to  1  mile . 8 

1  mile  to  1 ’A  miles . 7 

1  ’A  miles  to  2  miles . 6 

2  miles  to  2’A  miles . 5 

2'A  miles  to  3  miles . 4 

3  miles  to  3'A  miles . 3 

3'A  miles  to  4  miles . 2 

4  miles  to  5  miles . 1 


I  SEVERITY 

j  20  points  =  Major 
14-19  points  =  Moderate 
I  1 3  or  less  points  =  Minor 


The  causal  factors  attributed  to  the  ATCS  com¬ 
prise  a  hierarchy  of  specific  elements  within  more 
general  categories,  as  shown  in  Table  2.  For  example, 
if  a  computer  entry  had  been  incorrectly  updated 
(causal  factor  la.l)  then  that  error  also  involved  the 
less  specific  category  of  Data  Posting.  An  operational 
error  can  involve  multiple  causal  factors.  For  this 
study,  an  error  was  coded  as  1  if  a  factor  was  recorded, 
and  0  if  it  was  not.  T  able  2  also  includes  the  percentage 
of  the  1053  errors  that  involved  each  factor.  It  should 
also  be  mentioned  that  sometimes  the  more  global 
categories  were  not  recorded,  while  at  other  times, 
only  the  global  categories  were  recorded.  For  example, 
in  this  study,  if  computer  entry  error  was  reported  as  a 
factor  in  an  OE,  that  occurrence  was  also  coded  as 
having  involved  data  posting  as  a  causal  factor. 

Aircraft  Profile  Characteristics 

One  of  six  possible  combinations  of  aircraft  flight 
characteristics  was  identified  for  each  error  in  the 
sample.  The  profiles  were  as  follows: 


1)  all  aircraft  were  climbing, 

2)  all  aircraft  were  descending, 

3)  all  aircraft  were  at  level  flight, 

4)  one  (or  more)  aircraft  was  descending  and  one 
(or  more)  aircraft  climbing, 

5)  one  (or  more)  aircraft  was  at  level  flight  and  one 
(or  more)  aircraft  climbing, 

or 

6)  one  (or  more)  aircraft  was  at  level  flight  and  one 
(or  more)  aircraft  descending. 

RESULTS 

Factor  Relationships  to  0£  Severity 

Since  OE  severity  is  differentially  defined  for  aircraft 
above  or  below  29,000  feet,  the  following  results  are 
presented  in  terms  of  that  distinction.  As  shown  in 
Table  3,  the  average  number  of  aircraft  being  worked  at 
the  time  of  the  OEs  ranged  from  8. 10  for  minor  errors  at 
flight  levels  of  29,000  feet  or  below  (FL  290-)  to  10.59 
for  moderate  errors  at  flight  levels  above  29,000  feet 
(FL  290+).  The  number  of  aircraft  being  worked  was 


13 


Tabu  2 

Percentages  of  "Minor"  or  "Moderate"  Errors  at  En  Route  Facilities 
THAT  Involved  Each  Causal  Factor 


c.  Misunderstanding . 4 

d.  Readback . 20 

(1)  Altitude . 1 

(2)  Clearance  . 

(3)  Identification . 

e.  Acknowledgment . 

f.  Other . 

4.  Coordination . 1 


a.  Area  of  occurrence 

(1)  Inter-position . 

(2)  Intra-position . 

(3)  Inter-sector . 

(4)  Inter-facility . 

b.  An  aircraft  penetrated  designated 

airspace  of  another  position  of  operati( 
or  facility  without  prior  approval . 

c.  Coordination  was  effected  and 

controller(s)  did  not  utilize  informa¬ 
tion  exchanged  . 

(1)  Aircraft  identification . 

(2)  Altitudes/Flight  Level . 

(3)  Route  of  flight . 

(4)  Clearance  limit . 

(5)  Speeds . 

(6)  APREQS . 

(7)  Special  instructions . 

(8)  Other . 


1.  Data  posting . 13 

a.  Computer  entry . 6 

(1)  Incorrect  input . 2 

(2)  Incorrect  update . 1 

(3)  Premature  termination  of  data . 1 

(4)  Other . 2 

b.  Flight  progress  strip . 9 

(1)  Not  prepared . 0 

(2)  Not  updated . 3 

(3)  Posted  incorrectly . 0 

(4)  Reposted  incorrectly . 0 

(5)  Updated  incorrectly . 2 

(6)  Sequenced  incorrectly . 0 

(7)  Resequenced  incorrectly . 0 

(8)  Interpreted  incorrectly . 2 

(9)  Premature  removal . 0 

(10)  Other . 3 

2.  Radar  display . 59 

a.  Misidentification . 14 

(1)  Overlapping  data  blocks . 3 

(2)  Acceptance  of  incomplete  or 
difficult  to  correlate  position 

info . 1 

(3)  Improper  use  of  identifying  turn  ...  0 

(4)  Failure  to  reidentify  aircraft  when 

accepted  target  identity  becomes 
questionable . 1 

(5)  Failure  to  confirm  aircraft  identity 

after  accepting  a  radar  handoff ....  0 

(6)  Other . 1 1 

b.  Inappropriate  use  of  displayed  data  ..47 

(1)  Conflict  alert . 2 

(2)  Quick  look . 0 

(3)  ModeC . 12 

(4)  MSAW/EMSAW . 0 

(5)  Other . 37 

3.  Communications  error . 36 

a.  Phraseology . 3 

b.  Transposition . 5 


5.  Position  relief  briefing  deficiencies  noted  3 

a.  Employee  did  not  use  position  relief 

checklist . 1 

b.  Employee  being  relieved  gave 

incomplete  briefing . 1 

c.  Relieving  employee  did  not  make  use 

of  pertinent  data  exchanged  at 
briefing . 1 

d.  Other . 1 


Note:  0  indicates  <  .5% 
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Table  3 

Air  Traffic  Controller  Workload  and  Severity  of  Operational  Errors 


SEVERITY 


MINOR 

N 

MODERATE 

N 

#  of  Aircraft  Worked 

8.35 

737 

8.50 

315 

FL  s  290 

8.10 

487 

8.25 

281 

Fit.  Level  p<.001 

FL  >  290 

8.83 

250 

10.59 

34 

Severity  F^  ^'^=2.2;  NS 

Traffic  Complexity  Rating 

3.02 

737 

3.09 

315 

FL  s  290 

3.03 

487 

3.01 

281 

Fit.  Level  E,  i„g=.06;  NS 

FL  >  290 

3.06 

250 

3.29 

34 

Severity  F^  ^^i=.93;  NS 

Note;  FL  =  flight  level,  290  =  29,000  feet 


Table  4 

Severity  Classification  within  each  Complexity  Rating 


TRAFFIC  COMPLEXITY 

MODERATE 
SEVERITY  % 

MINOR 

SEVERITY  % 

FJ 

Easy 

27.2 

72.8 

103 

Below  Average 

29.9 

70.1 

197 

Average 

28.5 

71.5 

376 

Above  Average 

32.7 

67.3 

306 

Complex 

31.0 

69.0 

71 

Overall 

30.0 

70.0 

1053 

Note:  .89;  NS.  The  N  represents  the  number  of  OEs  that  occurred  in  each  complexity 

category. 


not  significantly  different  between  severity  levels 
(F,  ,p^j=2.2,  NS),  while  the  number  of  aircraft  being 
worked  was  significantly  greater  {F^  13.9,  ^.001) 

at  FL  290+.  The  average  air  traffic  complexity  rating 
was  not  significantly  different  between  the  severity 
categories  (f^,,o4*=-93,  NS)  or  flight  levels 
NS).  Similarly,  a  chi-square  test  suggested  no  significant 
difference  in  air  traffic  complexity  rating  across  OE 
severity  categories,  {A*^=1.89,  NS).  As  indicated  in 
Table  4,  the  percentages  of  errors  that  were  moderate 
(30.0%  overall),  compared  to  minor,  were  relatively 
consistent,  regardless  of  the  traffic  complexity. 
Specifically,  27.2%  of  the  OEs  that  occurred  under  the 
“easy”  complexity  rating  were  classified  as  moderate  in 


severity,  while  31.0  >  of  the  OEs  that  were  evaluated 
as  occurring  under  “complex”  conditions  resulted  in  a 
moderately  severe  OE. 

The  error  causal  factors  that  were  differentially 
related  to  OE  severity  are  shown  in  Table  5.  The 
number  of  cases  in  each  category  in  Table  5  equals  the 
total  number  of  errors  (both  minor  and  moderate  in 
severity)  that  involved  either  a  given  causal  factor  or 
ATCS  awareness  that  the  error  was  developing.  The 
percentages  represent  the  proportion  of  the  OEs  that 
were  moderately  severe  for  each  causal  factor.  The 
factors  that  were  associated  (determined  by  using 
A*, =^.01  as  the  criterion  level)  with  a  lower  percent¬ 
age  (compared  to  30%  of  all  OEs)  of  moderate  errors 
included:  a)  the  Misuse  of  Displayed  Dau  -  excluding 
use  of  Conflict  Alert,  and  b)  awareness  by  the  control- 
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Tabie  5 

Percentages  of  Errors  that  were  Moderately  Severe  by  Causal  Factor  and  ATCS  Awareness 


OVERALL 

FLs290 

FL>290 

% 

moderate 

N 

% 

moderate 

N 

% 

moderate 

N 

Factors  Involved  with  Greater  Severity 

Misuse  -  Conflict  Alert 

52.0 

25 

60.0 

20 

20.0 

5 

Communications 

36.9 

379 

45.0 

300 

6.3 

79 

Readback 

40.8 

206 

47.9 

169 

8.1 

37 

Readback  -  Altitude 

41.1 

151 

46.5 

127 

12.5 

24 

Coordination 

38.5 

161 

42.7 

110 

28.0 

51 

Inter-facility  Coordination 

53.8 

26 

60.0 

20 

33.3 

6 

Factors  Involved  with  Lesser  Severity 

Misuse  of  Displayed  Data 
(excluding  conflict  alert) 

22.0 

473 

27.9 

323 

9.3 

150 

ATCS  Aware  Error  was  Developing 

21.0 

267 

26.0 

177 

11.1 

90 

Note:  All  of  the  causal  factors  above  were  significant  at  ps.01  on  chi-square  tests. 
N=  the  number  of  OEs  that  were  related  to  each  factor. 

%=  the  percentage  of  each  N  that  was  moderate  (compared  to  minor)  in  severity. 
FL=  flight  level,  290=29,000  feet. 


Table  6 

Differences  in  Horizontal  Separation  Between  Aircraft  when  Operational  Error 
Involved  Certain  Causal  Factors  and  ATCS  Awareness 

HORIZONTAL  SEPARATION 


OVERALL 

FLs290 

FL>290 

MILES 

tl 

MILES 

MILES 

fm 

Factors  Involved  with  Greater  Severity 

Misuse  -  Conflict  Alert 

-.31 

25 

-.34 

20 

-.15 

5 

Communications 

-.25 

379 

-.28 

300 

-.16 

79 

Readback 

-.30 

206 

-.27 

169 

-.39 

37 

Readback  -  Altitude 

-.29 

151 

-.25 

127 

-.43 

24 

Coordination 

-.25 

161 

-.20 

110 

-.32 

51 

Inter-facility  Coordination 

-.42 

26 

-.43 

20 

-.41 

6 

Factors  Involved  with  Lesser  Severity 

Misuse  of  Displayed  Data 

.34 

473 

.33 

4 

323 

.37 

150 

(excluding  conflict  alert) 

ATCS  Aware  Error  was  Developing 

.28 

267 

.33 

177 

.13 

90 

Note;  The  values  presented  are  mean  differences  in  horizontal  separation  in  miles  for  errors 
that  involved  a  given  causal  factor  versus  all  other  errors.  Negative  values  represent  less 
separation  at  occurrence  while  positive  values  indicate  greater  separation. 

/V=number  of  OEs  that  were  related  to  each  factor. 
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Table  7 

Percentages  of  Errors  that  were  Moderately  Severe  by  Profile  of  Aircraft 


OVERALL 

FL<:290 

FL>290 

% 

MODERATE 

/V 

% 

MODERATE 

N 

% 

MODERATE 

N 

Profile  of  Aircraft 

All  Climbing 

21.9 

32 

23.3 

30 

0.0 

2 

Level  &  Climbing 

26.3 

323 

34.4 

218 

9.5 

105 

Descending  &  Climbing 

28.0 

75 

31.1 

61 

7.7 

13 

Level  &  Descending 

30.3 

469 

37.6 

346 

9.8 

123 

All  Descending 

34.5 

58 

36.5 

52 

16.7 

6 

All  Level 

43.0 

86 

52.9 

51 

28.6 

35 

Note: 

N=  the  number  of  OEs  that  occurred  in  each  aircraft  profile 
%=  the  percentage  of  each  N  that  was  classified  as  moderately  severe 
FL=  flight  level,  290=29,000  feet 


ler  that  the  error  was  developing.  Twenty-two  percent 
of  the  473  OEs  that  involved  Misuse  of  Displayed 
Data  resumed  in  a  moderately  severe  error,  and  simi¬ 
larly,  21%  of  the  267  OEs  in  which  the  ATCS  was 
aware  of  the  developing  situation  resulted  in  errors  of 
moderate  severity.  By  contrast,  other  causal  factors 
were  more  likely  to  result  in  moderate  severity.  In 
particular,  over  50%  of  errors  that  involved  Misuse  of 
Conflict  Alert  or  Inter-facility  Coordination  were  clas¬ 
sified  as  moderate.  Another  factor  related  to  a  greater 
loss  of  separation  was  Readback  (a  communication 
during  which  the  ATCS  fails  to  detect  a  pilot’s  incor¬ 
rect  response  to  a  clearance  provided  by  the  ATCS) 
and  more  specifically,  “Readback  involving  altitude 
information.”  When  Readback  was  a  causal  factor, 
over  40%  of  those  errors  were  classified  as  moderately 
severe. 

Table  6  lists  the  differences  in  average  horizontal 
separation  (overall,  and  at  FL  290-  or  FL  290+)  be¬ 
tween  those  errors  that  involved  each  of  the  causal 
factors  in  Table  5  versus  those  that  did  not.  For 
example,  for  the  errors  in  which  Misuse  of  Displayed 
Data  (excluding  use  of  Conflict  Alert)  was  involved, 
the  average  horizontal  separation  between  aircraft  was 
.34  miles  greater  than  the  OEs  that  did  not  involve  this 
causal  factor.  By  contrast,  in  the  OEs  in  which  Readback 


was  a  factor,  the  resultant  horizontal  separation  was 
.30  miles  less  than  the  OEs  that  did  not  involve 
Readback.  We  found  that  vertical  separation  was  not 
significantly  related  to  any  of  the  factors.  Thus,  the 
impact  of  these  factors  in  terms  of  OE  severity  cat¬ 
egory,  as  illustrated  in  Table  5,  was  found  to  be  related 
primarily  to  the  horizontal  separation  parameter. 

Table  7  examines  the  relationships  between  aircraft 
profile,  in  conjunction  with  flight  levels,  and  OE 
severity.  Tests  of  significance  were  not  conducted,  due 
to  low  expected  values  in  some  cells;  however,  the 
results  are  presented  for  descriptive  purposes.  Most 
OEs  in  the  overall  sample  occurred  when  one  or  more 
aircraft  were  level  and  a)  others  were  climbing  (N=525, 
31.0%),  or  b)  others  were  descending  (A^=469, 45.0%). 
Surprisingly,  the  greatest  likelihood  for  moderate  se¬ 
verity  occurred  when  all  aircraft  were  at  “level”  flight 
(43.0%  overall);  52.9%  at  FL  290-,  and  28.6%  at  FL 
290+.  Also,  flight  level  was  related  to  severity  for  each 
aircraft  profile;  i.e.,  a  greater  percentage  of  OEs  were 
moderately  severe  at  FL  290-,  compared  with  FL  290+. 


Table  8 

Average  ATCS  Workload  During  Moderately  Severe  OEs  at  En  Route  Facilities 


Facility 

N 

#  Aircraft 
Worked 

Traffic 

Complexity 

%  ATCS 
Awareness 

ZAB 

9 

10.00 

3.33 

(2) 

22 

ZAU 

31 

7.68 

2.77 

(1) 

0 

ZBW 

19 

5.79 

2.74 

(1) 

1 1 

ZDC 

35 

9.09 

3.71 

(2) 

23 

ZDV 

1  1 

9.27 

3.73 

(2) 

36 

ZFW 

16 

9.31 

3.31 

(2) 

6 

ZHU 

13 

8.92 

3.31 

(2) 

15 

ZID 

23 

9.30 

3.13 

(2) 

4 

ZjX 

20 

8.20 

2.50 

(1) 

20 

ZKC 

9 

10.00 

3.78 

(2) 

33 

ZLA 

15 

7.00 

2.60 

(1) 

20 

ZLC 

4 

1 1.25 

3.25 

(2) 

75 

ZMA 

1 1 

8.64 

3.09 

(2) 

18 

ZME 

10 

10.60 

3.30 

(2) 

10 

ZMP 

5 

8.60 

3.20 

(2) 

0 

ZNY 

26 

6.88 

2.77 

(1) 

31 

ZOA 

11 

7.45 

2.82 

(1) 

36 

ZOB 

19 

9.26 

3.00 

(2) 

21 

ZSE 

6 

7.33 

2.67 

(1) 

17 

ZTL 

9 

8.33 

3.78 

(2) 

11 

Overall 

Mean 

SD 

8.65 

1.35 

3.14 

.40 

21 

Note;  The  number  in  (  )  represents  a  classification  of  each  facility  as  either  relatively  high 
ATCS  workload  (2)  or  low  workload  (1)  reported  at  the  time  of  the  moderately  severe  OEs. 
Awareness=  the  percentage  of  each  facility's  moderately  severe  OEs  in  which  the  ATCS  was 
aware  that  the  error  was  developing. 


Facility-level  characteristics  of  moderately 
severe  OEs 

Tables  8  through  10  illustrate  the  results  of  analyses  by 
facility  for  the  moderately  severe  errors  at  ARTCC 
facilities  located  in  the  continental  U.S.,  i.e.  excluding 
ZSU,  ZHN,  and  ZAN.  Only  moderately  severe  errors 
(excluding  minor  OEs)  are  reported  here  because  of 
their  greater  salience  for  aviation  safety  and  our  finding 
of  nonsignificant  associations  between  workload  and 
OE  severity.  Table  8  lists  the  average  number  of 
aircraft  being  worked  and  air  traffic  complexity  rating. 
The  results  of  a  series  of  cluster  analyses  produced  two 
interpretable  groups  of  facilities  based  on  reported 
ATCS  workload.  Each  of  the  relatively  low  workload 
facilities  (ZAU,  ZBW,  ZJX,  ZLA,  ZNY,  ZOA,  and 


ZSE)  was  characterized  by  an  average  air  traffic 
complexity  rating  of  less  than  3.0  {“average”)  combined 
with  8.2  or  fewer  aircraft  being  worked.  It  should  be 
noted  that  some  degree  of  variability  could  be  expected, 
given  potential  differences  in  reporting  standards  and 
practices,  not  only  between  facilities  but  within  a 
facility  as  well.  T able  8  also  illustrates  facility  differences 
in  the  percentages  of  moderately  severe  errors  in  which 
the  ATCS  was  aware  that  the  error  was  developing. 
Specifically,  ZAU  and  ZMP  reported  that  no  ATCS 
was  aware  of  the  developing  error  in  any  of  their  OEs. 
By  contrast,  several  facilities  (ZDV,  ZKC,  ZLC,  and 
ZOA)  reported  that  the  ATCS  was  cognizant  of  the 
situation  prior  to  loss  of  separation  in  one-third  or 
more  of  their  moderately  severe  OEs. 
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Table  9 

Causal  Factors  Associated  with  Moderately  Severe  OEs  at  En  Route  Facilities  (%  of  Errors) 


FACILITY 

DISPLAYED 

DATA 

CONFLICT 

ALERT 

READBACK 

READBACK-  INTER-FACILITY 
ALTITUDE  COORDINATION 

ZAB 

22 

11 

33 

22 

00 

ZAU 

32 

03 

32 

26 

00 

ZBW 

32 

16 

21 

1  1 

05 

ZDC 

43 

00 

20 

14 

00 

ZDV 

27 

00 

27 

27 

00 

ZFW 

38 

06 

25 

25 

13 

ZHU 

23 

08 

46 

31 

00 

ZID 

35 

00 

30 

30 

09 

ZJX 

25 

15 

20 

20 

05 

ZKC 

11 

00 

44 

22 

00 

ZLA 

33 

00 

27 

13 

07 

ZLC 

100 

00 

00 

00 

00 

ZMA 

27 

09 

09 

09 

00 

ZME 

20 

00 

30 

20 

20 

ZMP 

40 

20 

00 

00 

00 

ZNY 

23 

04 

19 

15 

12 

ZOA 

27 

00 

45 

09 

18 

ZOB 

26 

00 

32 

26 

00 

ZSE 

33 

00 

33 

33 

00 

ZTL 

44 

00 

33 

11 

00 

Overall 

33 

05 

26 

18 

04 

Note:  The  data  are  the  percentages  of  a  facility's  moderately  severe  OEs  that  involved  these 
causal  factors. 


To  examine  ATCS  workload  in  conjunction  with 
awareness  that  the  error  was  developing,  multidimen¬ 
sional  scaling  (MDS)  was  applied,  with  the  results 
shown  in  Figure  1.  First,  the  two  workload  measures 
and  ATCS  awareness  data  were  aggregated  at  the 
facility  level.  Then  the  “2”  score  values  for  number  of 
aircraft  being  worked  and  traffic  complexity  were 
summed  to  compute  a  composite  measure  represent¬ 
ing  the  average  ATCS  workload  at  the  time  of  OE 
occurrence  for  each  facility.  A  dissimilarity  matrix 
among  facilities  was  created  based  on  F.uclidean  dis¬ 
tances  using  the  composite  workload  and  error  aware¬ 
ness  variables.  The  matrix  was  then  analyzed  using  the 
classical  MDS  approach,  in  which  the  cases  (en  route 
facilities)  were  plotted  in  two  dimensions.  Thus,  MDS 


provided  a  “geometric”  representation  of  the  facilities 
similarity/dissimilarity  in  terms  ofworkload  and  ATCS 
awareness.  Dimension  1  geometrically  represents  ATCS 
workload,  while  dimension  2  is  ATCS  awareness  that 
the  error  was  developing.  The  intersection  of  the 
dimensions  produced  four  quadrants  into  which  facili¬ 
ties  were  grouped,  based  on  their  relative  similarity/ 
dissimilarity  among  each  other.  Quadrant  1  was  de¬ 
fined  by  relatively  high  ATCS  workload  combined 
with  greater  than  average  error  awareness.  This  high 
workload-  greater  awareness  condition  was  illustrated 
best  by  ZDV,  ZLC,  and  ZKC.  The  high  workload-less 
awareness  facilities  (Quadrant  2)  included  ZFW,  ZTL, 
and  ZME.  ZAU  and  ZBW  were  characterized  by 
relatively  low  ATCS  workload  combined  with  less 


Table  10 

Profile  of  Aircraft  in  Moderately  Severe  OEs  at  En  Route  Facilities 


ALL 

DESCEND 

ALL 

LEVEL 

LEVEL 

ALL 

FACILITY 

CLIMB 

CLIMB 

DESCEND 

CLIMB 

DESCEND 

ZAB 

.0 

.0 

22.2 

.0 

77.8 

.0 

ZAU 

.0 

6.5 

.0 

22.6 

61.3 

9.7 

ZBW 

.0 

15.8 

5.3 

21.1 

52.6 

5.3 

ZDC 

2.9 

2.9 

5.9 

20.6 

58.8 

8.8 

ZDV 

.0 

.0 

9.1 

36.4 

45.5 

9.1 

ZFW 

.0 

.0 

.0 

37.5 

43.8 

18.8 

ZHU 

7.7 

.0 

23.1 

30.8 

30.8 

7.7 

ZID 

4.5 

13.6 

4.5 

31.8 

13.6 

31.8 

ZIX 

.0 

5.0 

5.0 

20.0 

60.0 

10.0 

ZKC 

.0 

.0 

.0 

55.6 

33.3 

1 1.1 

ZLA 

.0 

.0 

.0 

33.3 

46.7 

20.0 

ZLC 

.0 

25.0 

25.0 

.0 

25.0 

25.0 

ZMA 

9.1 

18.2 

.0 

27.3 

36.4 

9.1 

ZME 

.0 

.0 

.0 

50.0 

50.0 

.0 

ZMP 

.0 

.0 

20.0 

40.0 

20.0 

20.0 

ZNY 

3.8 

7.7 

7.7 

30.8 

46.2 

3.8 

ZOA 

.0 

20.0 

.0 

20.0 

40.0 

20.0 

ZOB 

5.3 

5.3 

5.3 

31.6 

42.1 

10.5 

ZSE 

.0 

.0 

16.7 

33.3 

50.0 

.0 

ZTL 

.0 

11.1 

11.1 

22.2 

44.4 

11.1 

Overall 

1.7 

6.6 

8.0 

28.2 

43.9 

11.6 

Note:  The  data  are  percentages  of  a  facility's  moderately  severe  OEs  that  occurred  under  the 
various  aircraft  profiles. 


awareness  (Quadrant  3),  while  ZNY  and  ZOA  re¬ 
ported  relatively  low  ATCS  workload  combined  with 
greater  awareness  (Quadrant  4). 

There  was  also  considerable  variability  in  the  fre¬ 
quency  of  the  causal  factors  previously  found  to  be 
related  to  severity.  For  example,  the  percentages  of 
OEs  that  involved  Misuse  of  Displayed  Data  ranged 
from  1 1%  to  44%  across  facilities  (Table  9).  Similarly, 
while  Readback  was  involved  in  26%  of  all  moderately 
severe  OEs,  at  the  facility  level,  between  0%  and  46% 
of  the  OEs  involved  Readback  as  a  factor. 

Table  10  illustrates  the  percentages  of  moderately 
severe  OEs  that  occurred  under  the  various  aircraft 
profile  categories  described  previously.  The  most  fre¬ 


quent  profile  -  aircraft  level  and  aircraft  descending  - 
characterized  over  40%  of  the  OEs  with  a  range  of 
13.6%  to  77.8%  across  facilities.  The  "All  level”  air¬ 
craft  profile  was  present  between  0%  and  31.8%  de¬ 
pending  on  facility. 

Overall  these  results  illustrate  why  future  research 
regarding  operational  errors  should  measure  facility- 
level  differences  to  estimate  the  extent  to  which  find¬ 
ings  are  generalizable  among  ARTCCs.  It  would  be 
interesting  to  investigate  the  relationships  that  may 
exist  between  these  facility-level  differences  and  facil¬ 
ity  differences  in  traffic  density,  local  procedures,  and 
airspace  configuration. 
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Figure  1 

Multidimensional  Scaling  of  En  Route  Facilities  based  on  Workload 
AND  Situation  Awareness  Characteristics  at  the  Time  of  the  OEs 


Note: 

Quadrant  I  =  high  workload  -  high  awareness 
Quadrant  II  =  high  workload  -  low  awareness 
Quadrant  III  =  low  workload  -  low  awareness 
Quadrant  IV  =  low  workload  -  high  awareness 

Workload  =  standardized  levels  for  number  of  aircraft  being  worked  plus  air 
traffic  complexity  at  the  time  of  the  OEs 
Awareness  =  ATCS  awareness  that  the  OE  was  developing 
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Table  11 

Frequency  of  Causal  Factor  Catecor'es  that  Resulted  in  "Major"  Errors  at  En  Route  Facilities 


c.  Misunderstanding . 1 

d.  Readback . 2 

(1)  Altitude . 1 

(2)  Clearance . . 3 

(3)  Identification . 1 

e.  Acknowledgment . 2 

f.  Other . 1 

4.  Coordination . 6 

a.  Area  of  occurrence 

(1)  Inter-position . 1 

(2)  Intra-position . 3 

(3)  Inter-sector . 2 

(4)  Inter-facility . 1 

b.  An  aircraft  penetrated  designated 

airspace  of  another  position  of 
operation  or  facility  without  prior 
approval . 1 

c.  Coordination  was  effected  and 
controller(s)  did  not  utilize 

information  exchanged . 5 

(1 )  Aircraft  identification . 1 

(2)  Altitudes/Flight  Level . 1 

(3)  Route  of  flight . 1 

(4)  Clearance  limit . 0 

(5)  Speeds . 0 

(6)  APREQS . 0 

(7)  Special  instructions . 1 

(8)  Other . 2 


1 .  Data  posting . 5 

a.  Computer  entry . 2 

(1)  Incorrect  input . 2 

(2)  Incorrect  update . 1 

(3)  Premature  termination  of  data . 1 

(4)  Other . 2 

b.  Flight  progress  strip . 4 

(1)  Not  prepared . 0 

(2)  Not  updated . 3 

(3)  Posted  incorrectly . 0 

(4)  Reposted  incorrectly . 0 

(5)  Updated  incorrectly . 2 

(6)  Sequenced  incorrectly . 0 

(7)  Resequenced  incorrectly . 0 

(8)  Interpreted  incorrectly . 1 

(9)  Premature  removal . 0 

(10) Other . 1 

2.  Radar  display . 9 

a.  Misidentification . 2 

(1)  Overlapping  data  blocks . 3 

(2)  Acceptance  of  incomplete  or 
difficult  to  correlate  position  info.  1 

(3)  Improper  use  of 

identifying  turn . 0 

(4)  Failure  to  reidentify  aircraft  when 

accepted  target  identity  becomes 
questionable . 1 

(5)  Failure  to  confirm  aircraft  identity 
after  accepting  a  radar  handoff  ....  0 

(6)  Other . 2 

b.  Inappropriate  use  of  displayed  data  ....  7 

(1)  Conflict  alert . 2 

(2)  Quick  look . 0 

(3)  ModeC . 1 

(4)  MSAW/EMSAW . 0 

(5)  Other . 7 

3.  Communications  error . 6 

a.  Phraseology . 3 

b.  Transposition . 1 


5.  Position  relief  briefing  deficiencies  noted  1 

a.  Employee  did  not  use  position  relief 

checklist . 1 

b.  Employee  being  relieved  gave 

incomplete  briefing . 1 

c.  Relieving  employee  did  not  make  use 

of  pertinent  data  exchanged  at 
briefing . 1 

d.  Other  . 1 
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Characteristics  of  OEs  of  Major  Severity 

Six  of  the  15  OEs  categorized  as  “major”  had  a 
complexity  rating  of  “above  average,”  while  four  were 
classified  as  having  occurred  under  “average” 
conditions,  two  had  a  “below  average”  rating,  and  3 
had  an  “easy”  rating;  producing  a  mean  complexity 
rating  of  2.87.  The  average  number  of  aircraft  being 
worked  was  8.47  with  the  range  between  2  and  15. 
These  findings  illustrate  the  sizable  variability  in  ATCS 
workload  conditions  found  during  the  OEs  classified 
as  “major”  in  this  study. 

Regarding  the  flight  characteristics  of  the  aircraft  in 
the  major  OEs,  three  of  the  OEs  occurred  at  FL  290+. 
In  five  cases,  all  of  the  aircraft  involved  were  in  a  level 
profile,  in  seven  the  aircraft  were  level  and  descending, 
in  one  OE  the  profile  was  level  and  climbing,  and  two 
occurred  when  aircraft  were  descending  and  climbing. 

Table  1 1  shows  the  frequencies  of  the  causal  factors 
that  resulted  in  OEs  that  were  rated  “major”  in  sever¬ 
ity.  The  Data  Posting  factor  was  involved  in  five  OEs, 
Radar  Display  in  nine.  Communications  in  six,  Coor¬ 
dination  in  five,  and  Relief  Briefing  in  one  OE.  (As 
noted  previously,  an  OE  can  involve  multiple  causal 
factors). 

DISCUSSION 
General  Findings 

A  counter-intuitive  finding  of  this  study  was  that 
neitl  zr  a  quantitative  measure  of  air  traffic  controller 
workload  (number  of  aircraft  being  worked)  nor  a 
rating  of  air  traffic  complexity  was  found  to  be  related 
to  OE  severity.  Future  research  could  examine  the 
correspondence  between  these  “objective"  workload 
measures  and  “subjective”  workload  assessments  made 
with  instruments  such  as  the  National  Aeronautics  and 
Space  Administration  Task  Load  Index  (Hart  & 
Staveland,  1 988)  in  order  to  expand  the  understanding 
of  the  workload  conditions  of  air  traffic  controllers. 

Causal  factors  that  relate  to  ATCS  awareness  would 
likely  include  those  that  are  not  easily  corrected  if  not 
caught  immediately,  allowing  the  error  situation  to 
build  until  it  is  too  late  to  avoid.  In  general,  the  causal 
factors  that  resulted  in  a  greater  loss  of  separation  were 
those  which  would  likely  reduce  awareness  of  the 
situation  by  the  air  traffic  controller  (i.e.  inter  facility 


coordination,  incorrect  readback  of  altitude  by  pilot, 
etc.).  This  assertion  is  supported  by  the  finding  that  if 
the  ATCS  was  aware  the  error  was  developing,  the 
errors  tended  to  be  less  severe.  Additionally,  the  misuse 
of  displayed  data  (excluding  conflict  alert)  was  associ¬ 
ated  with  less  severity,  which  suggests  that  the  factors 
associated  with  severe  errors  have  an  insidious  nature 
that  is  not  characteristic  of  errors  that  involve  the 
misuse  of  displayed  data. 

It  is  interesting  to  note  that,  for  each  factor,  only  the 
horizontal  separation  parameter,  not  vertical  separa¬ 
tion,  was  significantly  affected.  This  may  be  because 
altitude,  used  to  provide  vertical  separation,  is  re¬ 
ported  numerically  in  the  data  block.  Thus,  altitude 
information  is  likely  to  be  much  more  salient  than  the 
information  used  to  make  a  judgment  of  the  extent  of 
horizontal  separation  between  aircraft.  Judgment  of 
horizontal  separation  is  based  on  the  visual  distance  in 
separation  between  two  targets  on  the  plan  view  dis¬ 
play  and  judgments  of  their  relative  speed.  However, 
ATCSs  may  also  prefer  to  use  the  horizontal  parameter 
to  achieve  separation,  thereby  resulting  in  a  difference 
that  is  based  primarily  on  frequency  of  usage. 

Although  most  operational  errors  occur  when  at 
least  one  aircraft  is  in  level  flight  and  at  least  one 
aircraft  is  either  descending  or  ascending,  the  “All 
Level”  aircraft  profile  was  found  to  be  associated  with 
a  greater  percentage  of  moderately  severe  operational 
errors.  Perhaps  less  attention  is  directed  to  aircraft  in 
the  “All  Level”  profile,  resulting  in  a  greater  likelihood 
of  severe  errors.  Additionally,  moderately  severe  errors 
were  more  likely  to  occur  at  FL  s  290.  However,  at  FL 
5  290,  when  aircraft  are  in  an  “All  Level”  profile,  they 
typically  are  flying  at  faster  speeds  and  thus,  separation 
can  be  lost  more  rapidly,  potentially  resulting  in  a 
greater  loss  of  separation  between  aircraft. 

It  should  be  borne  in  mind  that  these  are  national 
averages  and  the  values  are  different  for  certain  facili¬ 
ties.  Differences  in  traffic  density,  local  procedures, 
and  airspace  configuration  may  limit  the  ability  to 
generalize  these  results  across  different  facilities.  This 
is  evidenced  by  the  results  of  the  facility  level  analyses 
presented  earlier.  Furthermore,  without  additional 
information  concerning  the  percentage  of  time  ATCSs 
at  the  various  facilities  spend  controlling  traffic  under 
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Two  facilities  best  exemplified  the  relatively  low 
workload/low  awareness  combination  (ZAU.ZBW), 
They  accounted  for  17%  of  all  the  moderately  severe 
errors.  However,  it  was  not  possible  to  determine  the 
extent  to  which  these  errors  were  a  result  of  traffic 
density,  airspace  configuration,  local  operating  proce¬ 
dures,  workstation  design,  poor  technique,  or  any  of  a 
number  of  other  factors.  In  fact,  since  normative 
information  is  not  available  on  variables  such  as  traffic 
density,  one  must  settle  for  a  description  of  the  factors 
associated  with  operational  irregularities. 


the  various  complexity  or  workload  conditions  it  is 
difficult  to  determine  the  primary  factors  associated 
with  these  outcomes. 

Facility  Level  Findings 

Although  neither  the  number  of  aircraft  being  worked 
nor  a  rating  of  air  traffic  complexity  was  found  to  be 
related  to  OE  severity,  a  series  of  cluster  analyses 
produced  two  interpretable  groups  of  facilities  based 
on  reported  ATCS  workload.  This  measure,  which  for 
low  ATCS  workload  was  characterized  by  an  average 
air  traffic  complexity  rating  of  less  than  3.0  (“average”) 
combined  with  8.2  or  fewer  aircraft  being  worked,  was 
used  to  examine  ATCS  workload  in  conjunction  with 
situation  awareness. 

Several  hypotheses  of  the  effect  of  workload  on 
situation  awareness  are  offered.  The  first  suggests  that 
awareness  decreases  as  workload  increases.  This  might 
be  due  to  difficulty  in  maintaining  an  accurate  mental 
picture  of  the  air  traffic  situation  as  its  complexity  and/ 
or  information  load  increases.  This  phenomena  is 
more  likely  to  operate  under  very  high  workload  con¬ 
ditions.  Another  possibility  suggests  that,  as  workload 
shifts  from  a  high  to  low  level,  awareness  decreases. 
This  situation  is  likely  to  exist  after  a  busy  traffic 
“push”.  Additionally,  awareness  may  decrease  if  low 
workload  conditions  are  sustained  for  an  extended 
period  of  time.  This  is  more  likely  to  occur  during  very 
low  workload  conditions.  Furthermore,  a  fatigue  ef¬ 
fect  could  be  operating  under  sustained  periods  of  high 
workload,  resulting  in  a  decrease  in  awareness  during 
later  periods  of  a  high  workload  watch. 

It  appears  that  reduced  situation  awareness  might 
occur  as  an  effect  of  very  different  workload  related 
causes.  In  this  research,  several  clusters  of  facilities 
were  distinguished  based  on  differences  along  the 
dimensions  of  ATCS  workload  and  situation  aware¬ 
ness.  Facilities  were  classified  as  either  high  workload/ 
high  awareness,  high  workload/low  awareness,  low 
workload/high  awareness,  or  low  workload/low  aware¬ 
ness.  The  data  are  not  available  to  determine  which  of 
the  four  hypotheses  mentioned  above  might  have  been 
operating  during  each  of  the  errors  analyzed  in  this 
study. 


“Major”  OE  Severity  Findings 

Fortunately,  over  the  31/2  years  of  data  analyzed  and 
presented  in  this  report,  only  1 5  of  the  1053  OEs  were 
rated  “major”  in  severity.  Causal  factor  frequencies  for 
the  “major”  errors  proportionately  matched  the 
frequencies  of  the  “minor”  and  “moderate”  causal 
factor  categories.  Additionally,  the  average  number  of 
aircraft  being  worked  did  not  differ  between  severity 
categories.  Furthermore,  the  profile  in  which  the 
majority  of  “minor”  and  “moderate”  errors  occurred 
(level  &  descending)  was  also  the  profile  of  the  majority 
of  the  “major”  errors.  Although  40%  of  the  “major” 
errors  were  rated  “above  average”  in  complexity,  the 
mean  complexity  for  the  “major”  errors  (2.87)  was  less 
than  that  for  the  “minor”  and  “moderate”  errors  (3.05). 
However,  given  the  small  sample  of  “major”  errors  this 
finding  was  not  statistically  significant. 


The  types  of  analyses  presented  in  this  report  allow  for 
the  identification  of  factors  more  likely  to  precipitate 
air  traffic  control  operational  errors.  However,  as 
mentioned  earlier,  due  to  the  lack  of  normative  data, 
one  must  settle  for  a  description  of  the  factors  associated 
with  these  operational  irregularities.  Additionally,  the 
reporting  process,  including  the  reporting  reliability  of 
the  investigators,  may  affect  the  extent  to  which  these 
relationships  can  be  determined.  For  the  investigative 
process  to  allow  for  a  more  definitive  determination  of 
the  factors  involved  in  operational  irregularities, 
investigators  must  be  able  to  review  the  dynamics 
associated  with  the  air  traffic  situation. 


CONCLUSION 
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Currently.  Quality  Assurance  (QA)  specialists  re¬ 
view  multiple  printouts  of  data  that  detail  aircraft 
locations  and  ATCS/Host  Computer  System  interac¬ 
tions,  as  well  as  multiple  audio  recordings  of  pilot/ 
controller  and  interphone  communications  to  deter¬ 
mine  events  surrounding  the  occurrence  of  an  opera¬ 
tional  irregularity.  The  task  of  piecing  together  the 
“big  picture"  from  multiple  sources  of  data  is  a  difficult 
and  time-consuming  part  of  the  QA  specialist’s  job. 
The  reliability  of  the  findings  of  the  QA  investigation 
affect  the  reliability  of  the  report  filed  for  each  irregu¬ 
larity.  To  allow  the  QA  team  to  spend  more  time 
identifying  the  sources  of  problems  associated  with  an 
incident  and  less  time  piecing  together  the  dynamics  of 
the  air  traffic  control  situation,  a  method  is  required 
that  allows  for  a  re-creation  of  incidents  and  integrates 
the  various  data  sources  into  a  coherent  picture.  Such 
a  system  has  been  developed  that  would  allow  for  the 
collection  of  normative  data  and  assist  in  determining 
more  definitively  the  location  and  likelihood  of  break¬ 
downs  in  the  human-machine  system  (Rodgers  & 
Duke,  1993). 

It  may  be  possible  to  quantify  characteristics  of  the 
air  traffic  control  situation  such  as  workload,  complex¬ 
ity,  severity,  as  well  as  others  as  a  by-product  of  the  re¬ 
creation  of  incidents  using  the  system  mentioned  above. 
This  would  reduce  the  variability  that  appears  to  exist 
between  investigators  and  facilities.  Standardization 
of  the  investigation  process  in  this  manner  would  allow 
for  an  improved  determination  of  the  factors  involved 
in  operational  irregularities.  Additionally,  such  a  sys¬ 
tem  would  provide  the  capability  to  collect  normative 
information  and  potentially  allow  for  more  than  a 
descriptive  view  of  these  occurrences. 
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APPENDIX  A 


Causal  Factor  Difinitjons* 

I.  Data  Posting  -  A  data  posting  error  is  any  error  of  calculation,  omission,  or  incomplete  data,  erroneous 
entries,  handling,  or  subsequent  revisions  to  this  data.  This  includes  errors  in  posting  and  recording  data. 
It  does  not  include  errors  involved  in  receiving,  transmitting,  coordinating,  or  otherwise  forwarding  this 
information. 

II.  Radar  Display 

A.  Misidentification  -  Radar  misidentification  means  a  failure  to  properly  identify  the  correct  target  and 
includes  subsequent  errors  committed  after  the  original  identification  was  properly  accomplished. 
Indicate  the  listed  item(s)  most  closely  describing  the  reason  for  misidentification. 

B.  Inappropriate  Use  of  Displayed  Data  -  A  data  or  display  information  error  occurs  due  to  a  failure  to 
maintain  constant  surveillance  of  a  flight  data  display  or  traffic  situation  and  to  properly  present  or 
utilize  the  information  presented  by  the  display  or  situation. 

III.  Communications  -  A  communications  error  is  a  causal  factor  associated  with  the  exchange  of  information 
between  two  or  more  people  (e.g.,  pilots  and  specialists).  It  refers  to  the  failure  of  human  communication 
not  communications  equipment. 

A.  Phraseology  -  Use  of  incorrect  or  improper  phraseology. 

B.  Transposition  -  Errors  due  to  transposition  of  words,  numbers,  or  symbols  by  either  oral  or  written 
means.  This  involves  writing/saying  one  thing  while  thinking/hcaring  something  else. 

C.  Misunderstanding  '  The  failure  to  communicate  clearly  and  concisely  so  that  no  misunderstanding 
exists  for  any  actions  contemplated  or  agreed  upon. 

D.  Readback  -  The  failure  to  identify  improper  or  incorrect  readback  of  information. 

E.  Acknowledgment  -  The  failure  to  obtain  an  acknowledgment  for  the  receipt  of  information. 

IV.  Coordination  -  Any  factor  associated  with  a  failure  to  exchange  requirement  information.  This  includes 
coordination  between  individuals,  positions  of  operation,  and  facilities  for  exchange  of  information  such 
as  APREQ’s,  position  reports,  forwarding  of  flight  data,  etc. 

V.  Position  Relief  Briefing  -  Relief  briefing  errors  arc  special  errors  of  both  communication  and  coordination 
that  occur  as  the  result  of  position  relief.  They  include  such  things  as  failure  to  give  a  relief  briefing,  failure 
to  request  a  briefing,  incomplete  or  erroneous  briefing,  etc. 

*  These  definitions  were  extracted  from  FAA  Form  7210-3. 
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